Chapter 11: Regular expression (RE)
From book Python Programming (Problem solving, Packages and Libraries) published by McGraw Hill Education (India) Private limited. By:

  • Anurag Gupta
  • G. P. Biswas

Note the following:-

  1. This html document is meant as an accompaniment to Chapter 11 Regular expression (RE) .
  2. The document contains scripts executed on IDLE as well as on Jupyter notebook.
  3. The scripts executed on Jupyter can be directly copied and run into a Jupyter notebook or some other IDE (Like Pycharm or Eclipse with PyDev or Visual studio).
  4. However the scripts on IDLE also contain the >>> symbol and therefore cannot be directly executed. If you want to execute them on IDLE or Jupyter, you need to manually remove the >>> symbol.
  5. Wherever needed some background material from the book is also included to help you better understand the scripts
  6. The topic numbers given on each paragraph match the topic numbers of the book, so you can easily identify the topics and corresponding scripts.
  7. In some of the scripts, the file paths give are that of the author's computer. You need to replace them with file paths of your own computer.
  8. At some places, to improve readability, page numbers of the book are indicated in green font like:- See Page 181 of the book
  9. This document was first created as a Jupyter Notebook as combination of Markdown and code cells (extension .ipynb) and then downloaded as html. If someone wants to "modify" or "extend' this document, you may ask for the original .ipynb file by sending me an e-mail at:- 999.anuraggupta@gmail.com

Note:- Topics 11.1 to 11.3 are mostly concepts and so are not covered here.
You can read then from the book

11.4 Understanding there` module
1. re.compile(some_pattern) and the Pattern Objects
See Page 249 of the book
In the compile() method, you use only the pattern and not the string in which the pattern is being looked for. So the re.compile(some_pattern) method takes only some_pattern as an attribute. Furthermore, the return value of compile(some_pattern) is what is called a pattern object. The advantage of using this is that if you are repeatedly looking for the same pattern in different strings, then you may compile it once only and use it repeatedly. Once you have a pattern object say p_obj, then you can use a method, such as p_obj.search(some_string), where some_string is the string being searched. Note here, that you will give only one parameter to search(some_string), namely the string str. So the method works in two steps.

  • First, you create a Pattern object using only some_pattern
  • Second, you use methods, such as match(some_string) or search(some_string) or other methods on this Pattern object you created. To summarize, the re module has a compile() function, which takes as its argument a “regular expression pattern”. The value returned by this function is a pattern object. Now the methods of this pattern object can be used for various pattern matching. This will become clear from the following example:
In [1]:
import re
myPat = '\w+'  # Create a pattern
myPatCompiled = re.compile(myPat)   # Compile the pattern
#Use findall() method of Pattern object. findall() returns a list
myWords = myPatCompiled.findall('Blowing in the wind')  
print(myWords)

['Blowing', 'in', 'the', 'wind']

You can check the type of the myPatCompiled object by running the following code on IDLE:

# ---ON IDLE---
>>> myPatCompiled = re.compile('\w+')
>>> type(myPatCompiled)
<class '_sre.SRE_Pattern'>

You can check the ‘type’ of object returned by the findall() method of the Pattern object by typing the following on IDLE:-

# ---ON IDLE---
>>> myWords = myPatCompiled.findall('Blowing in the wind')
>>> type(myWords)
<class 'list'>

2. re module convenience functions
See Page 250 of the book
As pointed out earlier, there are two ways to use the re module. The first was to create a Pattern object, which has been shown in the previous section.
The second technique is to directly use convenience functions. So, you don’t have to create a Pattern Object yourselves, but rather call the method of the Pattern object directly. Here, you have to give two attributes to the convenience functions.
The re module convenience functions are functions which take two parameters, namely some_pattern and some_string. Examples of these methods are re.match(some_pattern, some_string) where some_pattern is the pattern and some_string is the string on which the match is to be done. Note that the names of the “convenience functions” namely match() or search() are the same as in previous case (Where pattern object was created) but the “convenience functions” here take two parameters and not one (As was the earlier case when pattern object was created and then match() or search() were used with only 1 parameter that is some_string). This will become clear from the following example:

In [2]:
import re
some_pattern = r'\w+'
some_string = 'Blowing in the wind'
# Use re.findall(some_pattern, some_string) convenience function
myWords = re.findall(some_pattern, some_string)
print(myWords)   # Note findall() returns a list of words

['Blowing', 'in', 'the', 'wind']

The signature of the re.findall() function on Jupyter is as follows:

# ---ON IDLE---
Signature: re.findall(pattern, string, flags=0)
Docstring:
Return a list of all non-overlapping matches in the string. If one or more capturing groups are present in the pattern, return a list of groups; this will be a list of tuples if the pattern has more than one group. Empty matches are included in the result

` 11.5 The Match object and match(), search() methods.
11.3.1 The match() and the search() methods
See Page 251 of the book
The following code shows how match() method works:

In [3]:
import re
pat = "cat"
# t1 is a tuple of 3 raw strings
t1 = (r"cats", r"ca", r"a cat")
for s in t1:
    mobj = re.match(pat, s)
    print("\'cat\' in", s, "->", bool(mobj))

'cat' in cats -> True
'cat' in ca -> False
'cat' in a cat -> False

The following code shows the use of search() in a string. Suppose you want to search a simple pattern, say myPat = ‘cat’ in a string say myStr = ‘The cat is in the tree’. This can be done as follows:

In [4]:
import re
myStr ='The cat is in the tree'
myPat = 'cat'
myMatch = re.search(myPat, myStr)
if myMatch:
    print(myPat, ' <-found in ->', myStr)
else:
    print(myPat, ' not found in ', myStr)

cat  <-found in -> The cat is in the tree

We can show the type of Match object to be None if the search is not successful through the following code on IDLE:

# ---ON IDLE---
>>> text = "The cat is in the tree"
>>> p = 'dog'           # p has a pattern 'dog'
>>> m = re.search(text, p)  # Since pattern did not match returns None
>>> type(m)             # Check for type of return
<class'NoneType'>

This will be clear from the following two examples.
In example 1, first a pattern object is created and then from this Pattern object, a Match object is created.
In the second example, no Pattern Object is created at all. Rather, a Match object is created directly.
Example1:
See Page 254 of the book

In [5]:
import re
myPat = re.compile(r'boy')  # myPat is a Pattern object
myMatch = myPat.match('boy girl boy')   # myMatch is a Match object
if myMatch:
    print('pattern found')
else:
    print('Pattern not found')

pattern found

Example2:
The same code can also be executed directly by not creating a Pattern object but rather by directly creating a Match object as shown:

In [6]:
import re
myMatch = re.match(r'boy', 'boy girl boy')  # myMatch is a Match object
if myMatch:
    print('pattern found')
else:
    print('Pattern not found')

pattern found

In the following code (On IDLE), a pattern object and a match object are created and then their type is checked using the type() function:

# ---ON IDLE---
>>>import re
>>> myP = re.compile(r'boy')
>>> type(myP)   # myP is a pattern object.
<class'_sre.SRE_Pattern'>
>>> myM = myP.match('boy girl man')
>>> myM         # myM is a match object
<_sre.SRE_Match object; span=(0, 3), match='boy'>
>>> bool(myM)   # A match object if it exists is always bool True
True
>>> myM = myP.match('girl man')
>>> myM
>>> bool(myM)
False

11.3.2 Difference between match() and search() methods/ functions.
It is also important to know the difference between the re.match() and re.search() functions. The match() method only finds matches of the given pattern if they occur at the start of the string being searched. However, the search() method looks for a match not just at the beginning of the given string but looks for a match over the entire string. This will become clear from the following example:

In [7]:
import re
myMatch = re.match(r'girl', 'boy girl boy')  #myMatch is a Match object
if myMatch:
    print('pattern found')
else:
    print('Pattern not found')

Pattern not found

Consider the use of the search() function as shown:

In [8]:
import re
myMatch = re.search(r'girl', 'boy girl boy')  #myMatch is a Match object
if myMatch:
    print('pattern found')
else:
    print('Pattern not found')

pattern found

Note that the search() method of the Match object can be used to behave like the match() method by beginning the RE (Regular Expression) with a ‘^’ as shown on IDLE:

# ---ON IDLE---
>>> type(re.search("r", "abracadbra"))
<class'_sre.SRE_Match'>
>>> type(re.search("^r", "abracadbra"))
<class'NoneType'>
>>> type(re.search("^a", "abracadbra"))
<class'_sre.SRE_Match'>

Another example to explain the difference between match() and search() methods of the re module is as follows:

# ---ON IDLE---
>>> s = "abcd1234"
>>> m1 = re.match("[a-z]+",s)
>>> m1
<_sre.SRE_Match object; span=(0, 4), match='abcd'>
>>> m2 = re.match("[0-9]+",s)
>>> type(m2)
<class'NoneType'>
>>> m3 = re.search("[0-9]+", s)
>>> m3
<_sre.SRE_Match object; span=(4, 8), match='1234'>

11.6 Some important methods of the re module
The re module has some important methods, such as findall(), finditer() and sub(). They are discussed here.
(i) findall() will find all the instances of the pattern being searched.
(ii) The object returned by the findall() method is a list of the pattern found. This will become clear from the following example on IDLE:
See Page 257 of the book

# ---ON IDLE---
>>> myL = re.findall(r'boy', 'boy girl boy')
>>> myL
['boy', 'boy']# Note both “boy” found
>>> type(myL)
<class'list'>

11.6.3. re.sub(pattern, repl, string, count=0, flags=0)
The re module has a sub() method whose signature (On Jupyter notebook) is as follows:

Signature: re.sub(pattern, repl, string, count=0, flags=0)
Docstring:
(i) Return the string obtained by replacing the leftmost non-overlapping occurrences of the pattern in string by the replacement repl.
(ii)    repl can be either a string or a callable; if a string, backslash escapes in it are processed.  If it is a callable, it's passed the match object and must return a replacement string to be used.

This method can be used to replace parts of a string with the given pattern. This is best understood by an example (All digits are replaced by *):

In [9]:
import re
my_string = 'There were 50 people in 20 cars'
#Replace all digits with *
r1 = re.sub(r'\d', '*', my_string)
print(r1)

There were ** people in ** cars

11.7.1 Using group() method of the match object
See Page 259 of the book
The group() method returns one or more subgroups of the match. The possibilities are as follows:

  1. group() or group(0): Both are equivalent and return the entire matched substring in the form of a string.
  2. group(N): Returns the Nth matched subgroup in the form of a string. Note that the first subgroup is group(1) and not group(0).
  3. group( ... m, n, k, ...): This is a group function with multiple arguments. This will return a tuple of the subgroups. The number of items in the tuple are the same as the number of arguments to the group() method. For instance, group(1,3,4) will return a tuple of three items, namely the 1st, 3rd and 4th subgroups of the matched substring.
    Note: If the number of subgroups indicated as arguments to the group() method, are more than the number of subgroups in the Match object, there will be an error. This will become clear from following examples: 
# ---ON IDLE---
>>>matchObj = re.match(r'(spo*n) (spo*n)', 'spn spon spoon spooon')
>>>matchObj.group()
'spn spon'
>>>matchObj.group(0)
'spn spon'
>>>matchObj.group(1)
'spn'
>>>matchObj.group(2)
'spon'
>>>matchObj.group(1,2)
('spn', 'spon')

The following script is another example of how group() method can be used to extract parts of the substring. Remember that group() or group(0) will give the entire matched substring. But if you want only parts of this matched substring, then you have to use group(n) method where n is the number of the sub-group of the matched sub-string. This may appear a bit complicated, but is actually very simple and will become clear from following example on IDLE:

# ---ON IDLE---
>>>import re
>>> myP = re.compile(r"\d{2}-[a-zA-Z]*-\d{4}")
>>> myS = myP.search("15-August-1947 when India won freedom")
>>> myS.group(0)
'15-August-1947'
>>> myS.groups()
()
>>> myP = re.compile(r"(\d{2})-([a-zA-Z]*)-(\d{4})")
>>> myS = myP.search("15-August-1947 when India won freedom")
>>> myS.group(0)
'15-August-1947'
>>> myS.groups()
('15', 'August', '1947')
>>> myS.group(2)
'August'

11.7.2 Using start() and end() methods of the Match object
See Page 261 of the book
The general form of these methods is start(group_number) and end(group_number).If you have a Match object say matchObj, then matchObj.start(group_number) will give the index of the substring matched by the group and matchObj.end(group_number) will give the index of the end of the substring matched by the group. Note that if group_number is not given, or if it is 0 then it will default to the entire matched substring. Here, the case with no group number is considered, so the entire substring will be taken. This is best understood by the following on IDLE where group() is used on the match object. Further, the use of start() and end() methods of the match object are also shown as follows:

# ---ON IDLE---
>>> import re
>>> myStr = '012xxxxx89'
>>> matchObj = re.search(r'x+', myStr)
>>> matchObj.group()
'xxxxx'
>>> matchObj.start()    # Index of start ie first ‘x’ in ‘012xxxxx89’ is 3
3
>>> matchObj.end()
8   
>>> myNewStr = myStr[:matchObj.start()] + myStr[matchObj.end():] # All ‘x’  are being removed
>>> myNewStr
'01289'
>>> matchObj.span()
(3, 8)

11.7.3 The span() method of the Match object
See Page 262 of the book
The span() function returns a tuple, which contains the (start, end) positions of the match. Suppose you have a Match object, say matchObj. Then span() is equivalent to (matchObj.start(), matchObj.end()). Note that span()[0] is the same as start() and span()[1] is the same as end(). The syntax span()[0] may look a bit unusual, but it is actually quite valid because remember that span() is a sequence and therefore, indexable. This is clear from following script:

In [10]:
import re
myP = re.compile('\d+')
myI = myP.finditer('10 Ten 9 Nine 8 Eight 7 Seven')
for match in myI:
    print(match.span(),'From index',match.span()[0], 'to index', match.span()[1])

(0, 2) From index 0 to index 2
(7, 8) From index 7 to index 8
(14, 15) From index 14 to index 15
(22, 23) From index 22 to index 23

11.7.4 Greedy versus non-greedy matching
See Page 263 of the book
In general, a qualifier like * will try to match as much of the string as possible. Same is the case with {m, n}. The default behaviour is that a special character will try to match as much of the search sequence (string) as possible. This default behaviour is called Greedy Match. It is the normal behaviour of a regular expression, but sometimes this behaviour is not desired. To do this you can use a qualifier, that is, question mark, that is, ?. So the question mark, (?) is a greedy qualifier, that is, it qualifies a greedy behaviour into a non-greedy (also called lazy) behaviour. So expressions, such as *?, +?, ?? or {m, n}? will match as little of the string as possible. This will be clear from following example:

In [11]:
import re
str_html = '<!DOCTYPE html> <html> <head> </head> </html>'
print('length string->', len(str_html))
# 1---------GREEDY
pat_greedy = '<.*>'
print('length greedy->', re.match(pat_greedy, str_html).span())
print('groups greedy->', re.match(pat_greedy, str_html).group())
# 2---------NON-GREEDY
pat_nongreedy = '<.*?>' # has ? after * which makes it non-greedy
print('length non-greedy->', re.match(pat_nongreedy, str_html).span())
print('groups non-greedy->', re.match(pat_nongreedy, str_html).group())

length string-> 45
length greedy-> (0, 45)
groups greedy-> <!DOCTYPE html> <html> <head> </head> </html>
length non-greedy-> (0, 15)
groups non-greedy-> <!DOCTYPE html>

11.8 Some common scripts using RE
See Page 264 of the book
Write a script to find if the given substring is present in the string. This can be done in two different ways:
(i) By creating a Pattern object using the re.compile() function.

In [12]:
import re
inpPattern = input('Input the pattern to be matched-> ')
inpStr = input('Input the string on which match to be done-> ')
patCompiled = re.compile(inpPattern)    #Create Pattern object
myMatch = patCompiled.search(inpStr)    #Create Match object
if myMatch:                             #Must always test the Match object
    print(myMatch.group(), 'at ', myMatch.start(), 'to', myMatch.end()-1)
else:
    print('Not found')

Input the pattern to be matched-> abcdefghijklmnop1234abcd
Input the string on which match to be done-> 123
Not found

(ii). By directly using the convenience functions without creating a Pattern object, that is, creating the Match object directly:

In [13]:
import re
inpPattern = input('Input the pattern to be matched-> ')
inpStr = input('Input the string on which match to be done-> ')
#
myMatch = re.search(inpPattern, inpStr)    #Create Match object
if myMatch:                             #Must always test the Match object
    print(inpStr[myMatch.start():myMatch.end()], 
          'at ', myMatch.start(), 'to', myMatch.end()-1)
else:
    print('Not found')

Input the pattern to be matched-> 112
Input the string on which match to be done-> abcd11234112
112 at  4 to 6

(iii) How to create a list of words from a sentence using RE?

# ---ON IDLE---
>>> myS = "We Shall Overcome One Day"
>>>print(re.findall(r"\S+", myS))
['We', 'Shall', 'Overcome', 'One', 'Day']

(iv) Write a script to extract the server name, that is, gmail from the e-mail address abcd@gmail.com 

# ---ON IDLE---
>>> myP = re.compile(r"([a-zA-Z]+)@([a-zA-Z]+).([a-zA-Z]+)")
>>> myS = myP.search("abc@gmail.com")
>>>print(myS.group(2))
gmail

Beyond text book
See Page 268 of the book
2. Another topic which has not been covered in the text so far, is the concept of lookaround assertions. (Lookaround assertions consist of positive / negative lookahead and lookbehind assertions.). These four types are summarized in Table 1.4. of the book.
Consider the first , that is, a lookahead positive. Its syntax is (?= xyz), where xyz is some pattern. So if you have say a (?= t) it means that it will match only those patterns, where you have an a followed by a t.
Given below is a script, which uses a lookahead (positive) which matches a ‘,’ that is, a comma.

In [14]:
import re
# (?=,) is a lookahead positive pattern which matches all commas ie ','
# So \w+(?=,) will match all words ending with a comma
la_comma = re.compile(r"\w+(?=,)")
list_with_commas = la_comma.findall("A, B, and C, went to D, but not to E.")
print(list_with_commas)

['A', 'B', 'C', 'D']

Beyond text book Look around assertions
See Page 269 of the book
Some points to note about lookahead and lookbehind assertions (collectively called lookaround assertions ) are as follows:

  • Lookaround assertions are called zero-length assertions.
  • This is because they don’t consume any characters in the string.
  • Rather, they only give result of True, that is, a match or False, that is, no match.
  • This is why they are called zero length assertions. The term zero length means that no string is consumed and by assertion, the meaning is that it asserts whether the condition is True or False.

The following script shows how you can use a negative lookbehind regexp to select only those words which do not begin with an alphabet. So if you have a string, say, '300 men in5 !1 boats, 50 cats @2dogs' . Then the script should pick out only:- ['300', '1', '50', '2'] . This is shown in the following example:

In [15]:
import re
my_regexp = '(?<![A-Za-z])\d+'
regexp_compiled = re.compile(my_regexp)
test_string = '300men in5 !1 boats, 50 cats @2dogs'
digit_words = regexp_compiled.findall(test_string)
print(digit_words)

['300', '1', '50', '2']

11.9 Beyond text book:- Backreference 1
This topic is not covered in the book
Another topic that has not been covered in the chapter is what is called backreferencing. The method of Backreference allows one to match the same text, which has previously been matched by a capturing group. Before you understand the concept of backreference, you need to understand what parantheses, that is, ( and ) do in regexp. In regexp, parentheses serve the following purposes: (i) Parentheses ( ) can be used to create subexpressions. For instance, if you do (xyz)*, it means zero or more occurrences of pattern xyz.
(ii) Parentheses provide what are known as back-references. A back-reference means a reference to the matched substring.
We will explain the concept of backrefrence with the following example.
In the following script, backreference has been used to find duplicate words in a piece of text.

Regexp pattern is:-
r'(\b\w+)\s+\1'
The pattern has following 3 parts:-
(\b\w+) # (\b\w+) means a word
\s+     # \s+ means one or more white space(s)
\1      # \1 means a backreference to the first group in parentheses

So in effect, the regexp, r'(\b\w+)\s+\1' means a word followed by space and then followed by the same word again. You can use this regexp with backreference to remove duplicate words. The following script has a function remove_duplicates(), which removes consecutive duplicate words from a piece of text:

In [16]:
import re
def remove_duplicates(some_text):
    without_dup_text = re.sub(r'(\b\w+)\s+\1', r'\1', some_text)
    return without_dup_text
# Test the function
text_with_dup = 'The The cat cat sat sat on on the the wall.'
text_without_dup = remove_duplicates(text_with_dup)
print(text_without_dup)

The cat sat on the wall.

11.9 Beyond text book:- Backreference 2
This topic is not covered in the book
Another example of use of backreference is when backreference is used to swap two adjacent words. For instance, you can convert a string of type “cat dog” to “dog cat”.
Consider the following regexp:

reg_pat = r'([^\s]+)\s+([^\s]+)'
This regexp has 3 parts ie:-
1.  ([^\s]+)   # Matches all non-space characters
2.  \s+        # Matches space characters
3.  ([^\s]+)'  # Matches all non-space characters

The three parts are shown above. Out of the three parts, two are surrounded in parentheses. These two subexpressions surrounded in parentheses can be back-refrenced. The backreference can be done using the number of the subexpression starting from 1 (Not 0). So \1 will refer to the first subexpression, \2 will refer to the second subexpression, and so on.

In [17]:
import re
reg_pat = r'([^\s]+)\s+([^\s]+)'
old_text = 'ant bee cat dog'
new_text = re.sub(reg_pat, r'\2 \1', old_text)
print(new_text)

bee ant dog cat

It is possible to write a regular expression which checks for prime number. The script which does so is as follows (Using backreference):

In [18]:
import re
def is_prime(n):
    pat_for_prime = r'^1?$|^(11+?)\1+$'
    mo = re.match(pat_for_prime, "1" * n)
    if mo == None:
        return True
    else:
        return False
print('is prime 7->', is_prime(7))
print('is prime 9->', is_prime(9))

is prime 7-> True
is prime 9-> False